Finding Low-rank Solutions to Matrix Problems, Efficiently and Provably

A rank-r matrix X ∈ Rm×n can be written as a product UV >, where U ∈ Rm×r andV ∈ Rn×r. One could exploit this observation in optimization: e.g., consider the minimizationof a convex function f(X) over rank-r matrices, where the scaffold of rank-r matrices is modeledvia the factorization in U and V variables. Such heuristic has been widely used before forspecific problem instances, where the solution sought is (approximately) low-rank. Though suchparameterization reduces the number of variables and is more efficient in computational speedand memory requirement (of particular interest is the case r min{m,n}), it comes at a cost:f(UV >) becomes a non-convex function w.r.t. U and V .In this paper, we study such parameterization in optimization of generic convex f and focuson first-order, gradient descent algorithmic solutions. We propose an algorithm we call theBi-Factored Gradient Descent (BFGD) algorithm, an efficient first-order method that operateson the U, V factors. We show that when f is smooth, BFGD has local sublinear convergence,and linear convergence when f is both smooth and strongly convex. Moreover, for several keyapplications, we provide simple and efficient initialization schemes that provide approximatesolutions good enough for the above convergence results to hold.